Natural Dialogue in Modes Other Than Natural Language

Each command or transaction in a modern graphical user interface exists as a nearly independent utterance, unconnected to previous and future ones from the same user. This is unlike real human communication, where each utterance draws on previous ones for its meaning. While some natural language human-computer interfaces attempt to include such characteristics of human dialogue, they have been notably absent from graphical user interfaces. Our goal is to connect these properties of human dialogue to direct manipulation or graphical styles of user-computer interaction. This chapter describes our approach to incorporating some of the properties of natural dialogue (such as conversational flow, discourse, focus) into "other" (that is, not natural language) modes of human-computer communication. It also describes our research on building a natural user-computer dialogue based on a user’s eye movements. That work shows that starting from natural, rather than trained, eye movements results in a more natural dialogue with the user.